With the proliferation of capacitive touch screen devices, or motion tracking devices using infrared (IR) or devices based on image processing and camera, there is an opportunity for a graphics tool to create graphics intuitively based on free-sketch hand motions, just like it would be using pen and paper.
Free-hand drawing programs process sketch input received, whether through the user's input on a touch screen device, or through a motion-tracking device, and interpret the input as graphics. When the computer processes the free-hand sketch graphics, it can be processed as raster graphics like that of a physical ink and paper, i.e. color information is stored per location. But the sketch graphics information is hard to modify and not precise. In contrast, vector graphics organize it using logical concept, such as line and area and position. It allows users to easily modifiable and it is precise in representing the geometric primitives. It can also be extended for higher logical concept, such as the meaning of a diagram.
One of the most common and traditional approaches for handwriting recognition for converting free hand sketch to computer graphics is to matching the sketch with a list of predefined shapes. It can employ a wide variety of parameters for matching or different algorithm to find the best match. However, this approach is very inflexible, such that the result graphics is limited to the set of predefined shapes and it may not contain the variation and features desired by the user.
Another shortcoming of most tools is the process of conversion is very opaque to the user. A traditional free-hand drawing application attempts to “beautify” the sketch input by matching rough sketches to predefined geometric shapes, then replace the rough sketches with the matched predefined geometric shapes. Since it is hard to achieve 100% confidence to convert the free-hand sketching to the desired graphics by the user, it is important to have a good feedback system for a user to correct the conversion. And the feedback system has to be easy and intuitive, so that it wouldn't be more complicated than creating the graphics in traditional way.
The presented invention is an interactive system that converts users' hand-sketched drawing into the graphics that is intended by users. There are a few factors that caused the hand-sketched drawing to be different from the intended graphics. A good example would be that most people cannot draw a perfect circle on a piece of paper with a pen without the help of other tools. In the case of sketching on a touch screen with finger, there is noise and ambiguity in the system detecting the position of the finger, and the finger might block the user from seeing the position of the pointer.
United States Patent Publication 2006/0227140 Ramani et al., teaches a method for beautifying a drawn sketch (FIG. 2; Abstract); wherein there are implicit and explicit constraints (FIG. 1; [0108] and [0110]); changing lines to be either parallel or perpendicular based upon implicit constraints and satisfies the explicit constraints provided by user. See [0110]. Ramani discloses that an algorithm can be used to detect the implicit constraints and beautify the sketches ([0078]), wherein determining if the lines should be transformed to be parallel is based on the angle difference between the two lines ([0088]). Thus, Ramani teaches identifying a transformation between each pair of objects in the drawing to produce transformation information (determining the information of how the lines should be transformed (such as making the lines parallel) ([0088]).
However, Ramini does not disclose the process to prioritize a group of potential geometric constraints that can be prioritized based on their properties and be applied to the determined geometric primitives to produce a result with the highest priority that satisfies all the pre-existing geometric primitives and within the limit determined by the possible positional error.
In FIGS. 1A and 1B, it shows an example of the difference between the intended graphics and just the raw input of the user's hand. The present invention is designed to overcome the noise, ambiguity and other factor mentioned above through the use of geometric constraint solver, and the user's feedback to correct the result. It is based on the notion that most graphics can be composed of geometric primitives, e.g. lines, straight lines, curve line, circular arc, and geometric constraints, such as, parallelism and tangency. FIG. 2 shows a drawing with some of the geometric primitives and constraints pointed out-a straight horizontal line 20, a half dome circle arc 22, at least one connection between the circle arc 22 and the horizontal straight line 20, and five straight lines 26 emitting perpendicularly 24 from the circle arc 22.
By recognizing the right geometric primitives and right geometric constraint, it can recreate what the user intended, even given a very noisy input. And the present invention achieves this goal in a two pronged strategy—applying a reasonable set of geometric constraint with a novel use of geometric constraint solver, and providing an intuitive user feedback mechanism to correct any part of the result.